Quality determination for packetized information

ABSTRACT

A method for near real time quality analysis consistent with certain embodiments passively samples packets ( 616 ) from a stream of Internet Protocol (IP) packets that represent a communication session between a pair of end points carrying signals being transmitted over a transmission path in an IP network, and determines ( 620 ) in near real time at least two metrics from the sampled packets for the communication session. The at least two metrics include at least one metric that measures a quantity of lost packets, and at least one metric that measures a characteristic of packet timing. The method further involves calculating ( 632 ) a quality score in near real time using a quality formula that combines the at least two metrics. This abstract is not to be considered limiting, since other embodiments may deviate from the features described in this abstract.

CROSS REFERENCE TO RELATED DOCUMENTS

This application claims priority benefit of provisional patentapplication Ser. No. 60/483,781 filed Jun. 28, 2003, which is herebyincorporated herein by reference.

BACKGROUND

Historically voice telephone calls have been made using the PublicSwitched Telephone Network (PSTN). This networking environment has beendeveloped over the past hundred years using technologies that havecentered on making telephone companies more efficient through better useof existing wire and new fiber optic facilities. With data usage and theadvances in packet technology, Internet Protocol Telephony (IPT) is setto become the preferred networking method thus replacing traditionaltelephone environments.

The driving factors are compelling for both the economic and applicationvalue it brings to service providers, businesses and consumers. From aservice provider viewpoint, IPT significantly reduces infrastructure andoperational costs. These savings may be passed on to the customer andhelp the provider improve return on investment. From the customersperspective lower costs are an advantage, but perhaps more compelling isthe possibility of data and voice integration applications that were notpossible with traditional telephony.

Given the value of service provider and customer migration to IPT, it isnot surprising that research studies confirm there is pent up demand totransition to IPT. A key assumption supporting this demand is that thefundamentals of reliability and voice quality can be at least consistentwith, if not better than, the traditional telephone network. It islikely that customers will only move to IPT if the service levels andvoice quality of IPT meet these standards. The traditional providershave set a very high bar for uptime and voice quality, and consumershave come to expect close to perfection. The challenge to the IPTservice provider is to raise the bar that was set by matching voicequality and service levels then raise it through enhanced applicationservice offerings.

The current state of IPT testing is focused on network and carriertesting but is wholly inadequate for measuring the customer experience.

BRIEF DESCRIPTION OF THE DRAWINGS

Certain illustrative embodiments illustrating organization and method ofoperation, together with objects and advantages may be best understoodby reference detailed description that follows taken in conjunction withthe accompanying drawings in which:

FIG. 1 is a block diagram of a voice over Internet Protocol (VoIP)network.

FIG. 2 is a block diagram of a VoIP network with voice qualitymeasurement points indicated consistent with certain embodiments of thepresent invention.

FIG. 3 is another VoIP network configuration.

FIG. 4 illustrates the use of the RTP data in a manner consistent withcertain embodiments of the present invention.

FIG. 5 is an illustration of a VoIP quality measurement systemconsistent with certain embodiments of the present invention.

FIG. 6 is a flow chart illustrating a VoIP quality measurement processconsistent with certain embodiments of the present invention.

FIG. 7 is a screen shot illustrating one embodiment of a configurationmanagement screen consistent with certain embodiments of the presentinvention.

FIG. 8 is a screen shot illustrating one embodiment of a customer devicehistory summary screen consistent with certain embodiments of thepresent invention.

FIG. 9 is a screen shot illustrating one embodiment of a customersummary detail screen consistent with certain embodiments of the presentinvention.

FIG. 10 is a screen shot illustrating one embodiment of device historyscreen consistent with certain embodiments of the present invention.

FIG. 11 is a screen shot illustrating one embodiment of a another devicehistory screen consistent with certain embodiments of the presentinvention.

FIG. 12 is a screen shot illustrating one embodiment of point ofpresence (POP) summary screen consistent with certain embodiments of thepresent invention.

FIG. 13 is a screen shot illustrating one embodiment of a welcome screenthat defines the various types of screens a customer can encounterconsistent with certain embodiments of the present invention.

DETAILED DESCRIPTION

While this invention is susceptible of embodiment in many differentforms, there is shown in the drawings and will herein be described indetail specific embodiments, with the understanding that the presentdisclosure of such embodiments is to be considered as an example of theprinciples and not intended to limit the invention to the specificembodiments shown and described. In the description below, likereference numerals are used to describe the same, similar orcorresponding parts in the several views of the drawings.

The terms “a” or “an”, as used herein, are defined as one or more thanone. The term “plurality”, as used herein, is defined as two or morethan two. The term “another”, as used herein, is defined as at least asecond or more. The terms “including” and/or “having”, as used herein,are defined as comprising (i.e., open language). The term “coupled”, asused herein, is defined as connected, although not necessarily directly,and not necessarily mechanically. The term “program”, as used herein, isdefined as a sequence of instructions designed for execution on acomputer system. A “program”, or “computer program”, may include asubroutine, a function, a procedure, an object method, an objectimplementation, in an executable application, an applet, a servlet, asource code, an object code, a shared library/dynamic load libraryand/or other sequence of instructions designed for execution on acomputer system.

The term “near real time” is used in this document to mean that anaction (e.g., a calculation) is carried out at a time very close to anactual event so that the action is for practical purposes takenapproximately in real time. For example, a decoder that decodes a streamof data in near real time could receive the data, decode it and thenoutput the data in a manner transparent to the user. A non-real timedecoder might store the data in a file and then operate on the file tocarry out the decoding. Thus, by way of a contrasting example withoutlimitation, a non-real time action might involve storing information forretrieval at a significantly later time before an action is taken. Inthe context of the present discussion, near real time can be as long asseveral seconds or even several minutes, since the early stages ofdegradation of session quality can be detected and actions can be takenwithout significant impact to the customer and the customer experiencecan remain high. This is contrasted with non-near real time actionswhich might only provide insight into what happened in retrospect tocause call or session quality to seriously degrade, without the abilityto execute preemptive measures.

Telecommunications networks have gone through an evolution that hascreated a culture centered on a connection based network and a serviceprovider centric view of network management. A brief review of thishistory is instructive in understanding the evolution of IPT.

Initially all calls were carried on a single dedicated wire that wasconnected or cross connected by an operator on a switchboard. Once theconnection was made, the callers talked on a pair of wires transportinganalog signals from end to end. In this environment, degradation in callquality was determined primarily by signal loss caused by distance andthe number of connections frequently resulting in low volume and static.The voice quality solution in this environment was, in part, analogsignal regeneration and movement to automated switching centers.

As time and acceptance of the telephone progressed it became impracticaland problematic to continue stringing new phone wires to meet demand.This problem was solved by the development of technology that allowedmore than one phone conversation to be held simultaneously on a pair ofwires creating a multiplexed environment. First analog frequencies weresplit using frequency division multiplexing (FDM) and multiple callsover a single wire became possible. Using this method, voice quality wassomewhat degraded and alas demand outstripped the capacity and again anew method was needed.

The concept developed to resolve both these problems was theintroduction of digital signal technology. This technology converted theanalog spoken word into digital signals allowing high qualitytransmission of multiple phone calls over a single transport facility.The technology of pulse code modulation (PCM) was the first introductionof digital technology requiring special testing techniques to measurevoice quality. The signal conversion occurred in a coder, de-coder(CODEC). Voice quality testing was accomplished simply by convertingsignals back to analog using the same CODEC and measuring the analogsignal in the same manner the caller would hear the call—a simple buteffective solution.

This brief history of the migration from analog to digital in thetelephone network set the stage for a continuing need to gain more andmore efficiency in telephone company networks and the requirement tochange testing techniques to adapt to new technology and customer demandfor quality.

As technology advanced, additional efficiencies were gained through theintroduction of embedded signals in the call that caused the network totake action unrelated to the voice conversation being held. Thisintroduction of packet technology created a revolution in the telephonecompany's core network. The delivery of high quality analog signals wasthen relegated to the “last mile” (from the phone company's switchingcenter to end users).

In the 1990's customers began to adopt data applications at tremendousrates and the same capacity strains experienced with early telephonedeployments and a lack of facilities began to appear in the last milenetwork. This problem became severe in the late 1990's and was addressedby the introduction of competitive network builds in the local and longhaul facilities. These network builds again primarily incorporated thesame technology used in traditional systems.

In every instance traditional telephony has been able to replicate whatthe customer is hearing. This has been a cornerstone in maintaining thehigh level of service that customers expect today.

Traditional telephony is managed by shared responsibility between thecarrier and the customer. This shared responsibility requires a clearline be drawn between the two parties. The traditional phone companywill generally only accept responsibility to the last point where theyhave test capabilities from their central offices, beyond that point itis the customer responsibility to maintain and manage the environment.The result is a demarcation point that determines how the user handlestroubles or problems.

In the mid 1990's companies began to embrace the Internet and its coreprotocol (IP) as a preferred method to access data information. Theconvenience, availability, applications and relatively low cost createdcompelling reasons to begin using IP and the World Wide Web (WWW) as thedata transport methodology of choice. This environment uses packettechnology to transport and deliver information between networks andindividual desktop computers. It is highly efficient in the way it takesadvantage of associated but separate computer networks to accomplish itsmission of delivering information. Using voice transport technologiesinitiated by telephony needs over this data network has presented bothopportunities and problems. It has become possible to transport voicetelephone calls over IP (VoIP-voice over IP) and further to have voiceand data applications interact because of the use of a common protocol.The problems come from two primary areas: the differences in the waysnetworks are managed and the way in which voice quality is determined.

Any technology, be it VoIP or any other technology, that replacestraditional telephony transport and delivery should at least replicatethe voice quality of traditional telephony as gauged by the tried andtrue quality barometer: the human ear. To date, quality assurancetesting of IP Telephony has been data centric. However, monitoring andtesting methods are drastically different between voice and data.Therefore new methods of testing IPT, methods that are voice centric areneeded.

IPT and traditional telephony differ in many ways but have a common goalof delivering voice communication in real time between two or moreparties. The nature of IP is such that this delivery can be affected byvariables different than those that affect traditional phone calls. Oneof the biggest differences with IPT versus other techniques is thatanalog voice information can be converted into a data packet immediately(as early as within the phone itself) and can remain as data until itreaches the phone at the other end. Thus, measuring analog voicequality, as it is perceived by the caller, is fundamentally differentthan with traditional telephony which uses analog transmission oversubstantial distances.

Similar to PCM as outlined above, IPT relies on CODEC technology toconvert information from analog to digital signals and compress theinformation. But unlike PCM, the CODEC adds network-signalinginformation at the customer premise—perhaps even at the end user'sdesktop. Add to this a high dependence on the customer local areanetwork (LAN) and its equipment infrastructure, and what you end up withis an extension of a network that was once the sole domain of theservice provider or telephone company, and that now incorporates thecustomer's own LAN and all its vagaries.

One example of a basic IPT system 100 is depicted in FIG. 1. In thisnetwork, speech from a user at 102 is converted to packetized IPprotocol data by an IP telephone 106. This data are passed through anappropriate switch 108 along with data from other IP phones such as 110and 112 and perhaps a firewall 114 to a router 116 and a VoIP gateway120 where the packets enter an Internet Protocol network 124 (i.e., theInternet, or an IP protocol LAN (Local Area Network) or WAN (Wide AreaNetwork). The packets are routed through the IP network 124 and emergeat a similar arrangement on the receiving side. The packets first entera VoIP gateway 130 and pass through an appropriate router 134 throughfirewall 136 to a switch 140. The switch 140 passes the packets to thedestination IP telephone 144 (or IP phones 146 and 148, etc.) where thepackets are converted to analog audio signals for listening at 150 bythe recipient. At the recipient end, a similar process takes audiooriginating at 154 through the same network in the opposite direction toemerge at the other side at 160.

In a network such as network 100 (or any IPT network), a number offactors can affect IPT voice quality. Keeping in mind that IP wasoriginally designed and optimized for data traffic, the network'scharacteristics are specifically not optimized for providing highquality voice. In the data environment, information is generally nottime sensitive, has little concern for lost bits of the informationstream (as it can be retransmitted) and is not particularly affected byvariations in the packet-to-packet delivery timing. Voice, on the otherhand, can be adversely affected by any or all of these issues.

For purposes of this document, several characteristics of the transportof packets are of particular interest. The terms for thesecharacteristics will be defined as follows, with the definitions refinedlater: The term “latency” as used herein, is the time it takes for avoice packet to leave the originating end and arrive at the destination.High latency with respect to voice usually results in an echo-likeeffect. The term “Packet Loss” as used herein refers to information thatis sent from a source point A and is not delivered to the intendeddestination point B, for whatever reason. Packet loss can cause anunintended clipping, choppiness or silence during a call. The term“Jitter”, as used herein, is the variation in the delay time from onepacket to the next. In traditional telephony, all network components arecarefully timed by a master clock keeping each piece of information instrict time sequence. With IPT, gateways, routers, switches andfirewalls make mostly individual decisions as to when to forward voicepackets. This can result in this variation in packet-to-packet deliverytime causing the call to have a “warble” effect.

While it is relatively easy to adjust network equipment to ensureoptimization for voice, it is difficult to identify when and where aproblem exists, quantifying the voice quality degradation, anddetermining what factor or factors are causing the voice qualitydegradation.

CODECs used in IPT are often quite adept at adjusting to most qualityimpacting events. Where they often have pronounced difficulty is in“understanding” the effect on the listener when one or more qualitydegrading events occurs. Simply monitoring latency, packet loss andjitter will not necessarily tell the service provider what the caller isexperiencing.

There are a number of causes of poor voice quality. The causes of poorvoice quality can be attributed to almost any piece of network equipment(active devices) that acts upon the voice packet information or thetransport network itself. IPT is highly dependent on logical transportand route management where traditional telephony is generally affectedmore by physical transport management. The result of these differenttransport management methods is that IPT voice quality management is farmore critical than traditional telephony because of the greater numberof systems and routing options in IPT than than traditional telephony.

IPT uses statistical management to determine voice packet forwarding androuting. By its nature statistical management makes decisions on whichpackets go at what time and to whom. During times of high traffic load,the packet processors need to make critical decisions on what voiceinformation to send and when to send it. This means that inevitably somepackets will be dropped or lost. As stated earlier, this is not criticalfor data, because data can be resent. But when information is dropped inthe middle of a voice call, the information cannot be resent. The netresult is a “chopping” or “clipping” effect heard by the listener. Poorcongestion management can thus result in poor voice quality.

Errors occurring on the physical transport facility can cause the sameeffect as congestion. If an error is taken on a physical transportfacility the entire voice packet can be lost. In traditional telephonymost transport will be unaffected by low error rates on the physicaltransport facilities.

In all the router and gateway environments that make up pieces of an IPTnetwork, decisions are made giving priority of some packet informationover others. If prioritization is not optimized for voice traffic, delaycan occur producing loss of packets jitter or latency as a resultcausing poor voice quality.

The ability of the network equipment to process packet information in atimely manner is important to moving packets quickly to the nextdestination. Processor load can cause delay in forwarding packets andalso result in jitter, latency or packet loss, causing poor voicequality.

In most packet networking equipment, information is stored in memoryprior to forwarding to the processor. If memory becomes full oroverloaded, voice packets can be dropped causing voice packet loss andpoor voice quality.

In order for the IPT telephone experience to achieve high quality, theexperience should be appropriately managed. Fundamentally, callerexperience is the same with IPT as with traditional telephony. The usergenerally does not care what mechanism is used to transport thetelephone call. It is only important that the call be clear andreliable. Because the ear is an analog hearing device, an importantpoint of measurement is where the digital information is converted toanalog. But unlike traditional telephony, it is either impossible orimpractical to “plug in” a testing device to determine the quality of acall. Therefore, the digital information should be measured in a waythat mimics what the human ear processes, and/or detects parameters thatcan be translated to call quality.

FIG. 2 depicts a network 200 similar to that of FIG. 1, exceptillustrating that various metrics of voice quality can be measured usingan appropriate voice quality measurement device 210, at any of numerouspoints in the network prior to entering the IP network 124. In someinstances, the IP network 124 itself, in whole or in part, may also bemeasured to provide useful metrics related to quality.

With end-to-end IPT it is possible and generally most desirable todeliver the pure IP call information as close to the caller as possible.This allows for deep voice and data integration to the user phone anddesktop computer. Certain partial IPT implementations, as illustrated bythe network 300 of FIG. 3, convert digital to analog near or at the edgeof the customer network rather than at the end user desktop or handset.This is done either within the service provider network or at thecustomer private branch exchange (PBX) in order to lower customer usagecosts. A partial implementation, however in most instances, makes itdifficult or impossible to get the optimal value in IPT.

In network 300, an analog telephone 304 takes input from 102 anddelivers analog signals to the PSTN (Public Switched Telephone Network)310 which provides analog signals to a VoIP gateway 320. In this system,the signal is analog until reaching the VoIP gateway which coverts theanalog signals to IP protocol packetized data that is passed over the IPnetwork 124 to a destination VoIP gateway 330. At 330, the digitizedpackets are converted back to analog where they are passed through thePSTN 340 to an analog telephone 346 for the user at 150. In a similarmanner, speech originating at 154 and destined for 160 is processed inthe reverse direction.

In order to ensure that testing monitors the actual customer experience,monitoring a session from CODEC to CODEC may be the most (and possiblyonly) valid testing point. The challenge comes because it is generallyimpractical to install dedicated testing systems at each CODEC orcustomer phone. Thus, to resolve this issue, testing methods shouldpreferably be developed without dedicated equipment at each CODEC endpoint, but that still produce CODEC to CODEC test results.

Perceptual measurement techniques such as PSQM (Perceptual SpeechQuality Measurement) and PESQ (Perceptual Evaluation of Speech Quality)measure the difference between a reference analog signal and a degradedanalog output. These techniques use a known reference, usually astandardized recorded phrase in order to accurately measure a call. Theygenerally use a controlled environment and outboard testing systems.Testing using PSQM and PESQ have become preferred methods during networksetup and general network failures. Although accurate, these methods areimpractical at customer locations due to the cost of testing systems andthe intrusive nature of the test. These are best suited for use withinthe service provider network.

Many service providers attempt to avoid the potential problems of voicequality by overbuilding the transport and network components. Thisenvironment, although initially effective, only masks the potentiallong-term problem and reduces the value of IPT because networkefficiency is not maximized. With the current abundance of capacity thismethod is initially attractive but does not effectively prepare thenetwork environment for the inevitable need to maximize utilization.

Some providers and users resort to reactive management based on customercomplaint. Once a caller complains, an engineer or technician can drawstatistics from different network elements and deduce suspected causes.Then, largely by trial and error, corrections are made to the network.This is highly undesirable in an environment where callers expect highquality or perfection on every call.

In accordance with certain embodiments consistent with the presentinvention, actual telephone calls are measured, preferably from end toend, in order to provide an effective measurement of the callerexperience. In this manner, measurements can be effective in assuring acaller experience is on par with traditional telephony.

Certain embodiments consistent with the present system results in thefollowing preferred environment:

-   -   Every call or session (or at least a representative sample) is        preferably measured    -   Testing is preferably CODEC to CODEC, Caller to Caller    -   The actual caller experience is preferably evaluated and        reported    -   Call quality events and problems are preferably isolated to        specific network sections and components    -   Management information is preferably presented in near real time

In accordance with certain embodiments consistent with the presentsystem, actual calls (or other multimedia communication sessions) areused to evaluate the caller experience. Information is derived from thepacket stream of the call and applied to an algorithm that assigns ascore to the call.

If the call's score is outside predetermined acceptable limits,proactive measures can be taken to improve call quality. In most cases,such measures can be taken before the caller even perceives thedeviation in quality that caused the proactive measures to be taken. Incertain embodiments, this provides a substantial enhancement to existingVoIP technology since actions can be taken proactively to correctproblems, often before they are noticed by the customer, and thusproviding a mechanism to provide enhanced VoIP service quality.

This approach derives information from the real-time transport protocol(RTP) information of the voice packet as illustrated in the packetdiagram 400 of FIG. 4, and applies a calibrated formula to determine aquality score of the call. The RTP data are shown as 404. By makingcalculations from embedded data within the IP packet, tests arecompletely non-intrusive and passive.

With reference to FIG. 5, in certain embodiments consistent with thepresent invention, a Call Quality Analyzer (CQA) 500 is embodied assoftware running on a programmed processor (such as a computer), thatprovides a real-time view of conditions affecting voice quality on aVoice-over-IP (VoIP) network. The software can reside on one or morecomputer systems on the network, and collects data from one or moresources, including (but not limited to):

-   -   Samples of network communications, in which digitized voice or        other multimedia is carried in Real Time Protocol (RTP) packets    -   Metrics contained within Real Time Control Protocol (RTCP)        packets    -   Network Performance Test Probes (NPTP) such as Service Assurance        Agent (SAA) probe results and their equivalent (all equivalent        and similar such results are referred to herein as NPTP results        without limitation) such as those recorded on a Cisco brand VoIP        gateway or other network device    -   Call metrics stored on a softswitch or other VoIP (or multimedia        IP) network element    -   Call metrics stored on a handset or other end-device

Thus, in accordance with certain embodiments consistent with the presentinvention, any number of call metrics can be leveraged to produce a callquality score that can be used to maintain high call or session quality.

As depicted in FIG. 5, CQA 500 uses a passive stream collector 504 tosample a stream of packets 508 in the network. These packets 508 couldbe passively sampled at any number of places, but in the illustratedembodiment, the samples are taken at the input and/or output of switch108 (e.g., in network 100) in order to most closely sample the voicequality from CODEC to CODEC.

When the term “passive” is used in connection with the passive streamcollector 504, the term is intended to mean that the stream collector504 simply reads passing packets without disturbing them. The streamcollector 504 does not block, route, delay, relay or in any other wayinfluence the packets it is collecting (no effect from source todestination). In other words, collector 504 simply acts in the capacityof an observer that has no effect on the traffic it is observing. Thus,the passive stream collector 504 does not operate as a part of an activeelement of the network and is completely non-intrusive to an existingnetwork. In certain preferred embodiments, the passive stream collector504 is implemented as a software process residing on a server that isseparate from any of the active network components that actually handlemovement of the data traffic (e.g., phone, gateway, switch, router,etc.).

In certain preferred embodiments, the passive stream collector 504operates on a server that preferably is physically situated near aswitch. This enables CODEC to CODEC testing of analog signal quality.The passive stream collector 504 processes whatever packets it sees. Ifit sees packets coming from an IP telephone, it processes those packets.If it is connected to a switch to process all packets to and from theswitch it processes those packets. The passive stream collector 504should preferably see at least all of the signaling and media packetsfor at least one direction of an entire call or session. In a preferredimplementation, the passive stream collector is attached to a switch(e.g., 108 and/or 140) which redirects copies of all packets goingthrough the switch to the passive stream collector 504. Otherimplementations are also possible within the scope of embodimentsconsistent with the present invention.

The packet stream samples are then analyzed to determine values forpacket loss (P) at 514, latency (L) at 518 and jitter (J) at 522 and/orother packet timing information. This data are then plugged into aquality formula at a stream quality analyzer 530. Stream qualityanalyzer, in the current embodiment, is a programmed process operatingon a computer that may reside at a centralized remote location such as acall center or may reside on the same computer used to implement passivestream collector 504 or may reside on another computer withoutlimitation. If additional data are available such as NPTP results 534,soft switch call metrics at 538, call metrics from handsets or otherend-devices 540, and other metrics at 544, this data may also beutilized in the quality formula to produce a quality score output at550.

The passive stream collector 504, it should be noted, can also samplemultiple streams (sessions, etc.) simultaneously using the same instanceof the stream collector 504, in accordance with certain embodiments. Thesetup and teardown of distinct sessions can be determined by informationin the signaling protocol that can be recognized by the passive streamcollector 504. The quality score can then be generated on a session bysession basis so that each score is associated with a pair of end pointscarrying out the session (e.g., a VoIP telephone call, or othermultimedia communication session).

This quality score (Q) can then be used in a number of ways to eithermanually or automatically take actions to assure that the call qualityis at an acceptable level. For instance, the quality score can bedisplayed at 560 for monitoring by an operator (The values of P, L, Jand other metrics may also be displayed.). The data can also be storedfor later use or for use in refining the quality score equation at 564.In one embodiment, the quality score is associated with the two endpoints for the session and stored in a database indexed to these two endpoints for later use.

When the quality score is displayed, it can be displayed in any numberof manners. For example, the display can be generated using a web basedor application specific computer program. A grid or various coloringschemes can be used to represent various thresholds or judged qualitylevels, historical data can be presented for the particular pair of endpoints, etc. Since multiple sessions can be monitored by the passivestream collector 504, the display can present a display based upon theend point pairs, on a single end point device or on a site or otherbasis.

The score can also be used to generate an alarm or alert at 568 bycomparison of the score with one or more a thresholds to assure thatproper intervention is taken if the call quality degrades belowacceptable levels. Thus, for example, if the call quality score is madeto track the PSQM scale, where a score of 3.4 represents barelynoticeable distortion, an alert can be set whenever the call qualitydegrades below 3.4 or alternatively, another threshold can be set aboveor below this threshold as desired as will be discussed in greaterdetail later. Such thresholds can be determined empirically based uponlistening tests or other mechanisms. In this example, larger numbersrepresent poorer quality, while on other scales smaller numbers may bemade to correlate to poorer quality. In either case, when the thresholdis crossed in the direction of poorer quality, the quality can be saidto fall “below the quality threshold.”

In addition, the quality score can be used to effect alternate routingsusing routing control mechanisms at 572. Also, the individual deviceparameters within the network can be adjusted at 576 to assure thatchanging loads, faulty equipment and other variables are properlyaccounted for to provide optimum call quality.

In certain embodiments, it may be advantageous to aggregate a set ofquality score samples to assure that an anomaly within a set of samplesdoes not needlessly cause an alarm or alert or network componentreconfiguration.

Thus, a near real time quality analyzer consistent with certainembodiments has a passive stream collector that samples packets from astream of Internet Protocol (IP) packets that represent a communicationsession between a pair of end points carrying analog signals beingtransmitted over a transmission path in an IP network, and determines innear real time at least two metrics from the sampled packets for thecommunication session. The at least two metrics can include at least onemetric that measures a quantity of lost packets, and at least one metricthat measures a characteristic of packet timing. A stream qualityanalyzer receives the at least two metrics and calculates a qualityscore in near real time using a quality formula that combines the atleast two metrics. In accord with certain embodiments, the at least onemetric that measures a characteristic of packet timing measures at leastone of packet jitter, packet latency, and round trip time.

As previously noted, the CQA is not an active element of the VoIP orother multimedia network; that is, it does not participate in signaling,or otherwise directly interfere with the setup, progress,transportation, or teardown of VoIP or other multimedia sessions. Itmay, however, provide input to other VoIP or multimedia network elementsthat may affect VoIP or multimedia sessions based on the input (forexample, by rerouting calls to higher-quality links).

From the data collected by the CQA 500, the values of basic networkmetrics such as jitter, packet loss, and latency are determined forongoing VoIP or multimedia sessions and/or for links between elements ofthe VoIP network. These values are then inserted into a formula thatgenerates a numerical quality score characterizing the fidelity of vocalcommunications during the session or carried over the link. Scores aregenerated frequently and preferably made available for display (forinstance, from a web server) or further processing.

The formula is not necessarily a steady state formula. It is determinedby calibration of each deployment and varies based upon the equipmentconfiguration primarily, but not exclusively, based on the IP device(i.e., the IP telephone or IP video appliance, etc.). Recalibration ofthe formula may be needed when changes are made to the network.

One system strength of certain embodiments is the ability to create aseries of formulas for different CODECs and devices. That is, if aManufacturer X phone is using CODEC 1, and a Manufacturer Y phone isusing CODEC 2, then the perceived voice quality to the end user can bescored on each device using different formulae based on prior knowledgeof the type of device at each end, all of which can be done in the samenear real time manner.

In one embodiment, the formula is determined by correlating samples ofnetwork metrics (jitter, packet loss, latency) to empirical observationsof quality. These observations may be subjective (e.g., ratings by humanlisteners) and/or objective (e.g., scores produced by computerized voicequality testers). The formula is calibrated to generate quality scoresthat should closely reproduce the results of empirical observations. Therange and distribution of scores can be made to correspond to anycommonly used voice scoring system, such as for example Mean OpinionScore (MOS), PSQM, PESQ, Measuring Normalized Blocks (MNB), or R factor(transmission Rating factor), so that the meaning of the numericalscores will be evident to those familiar with the scoring system.

Evaluating whether a telephone call is good or bad is highly subjective.The uniqueness of the human ear and the ability of a listener to discernsound variations make the objective, consistent measurement of callquality difficult. The first voice quality measurement system wasdeveloped by the ITU in the mid-1990s under the P.800 standard, “Methodsfor Subjective Determination of Voice Quality.” The output of P.800, theMean Opinion Score (MOS), is calculated by having a group of male andfemale listeners in a controlled environment, subjectively rate a seriesof audio recordings. MOS is scored on a 1 to 5 scale, with 4.0 andhigher considered toll quality. This rating system is summarized inTABLE 1 below: TABLE 1 MOS Rating Call Characteristics Listening Effort5 Excellent imperceptible distortion complete relaxation, no effortrequired 4 Good just perceptible attention necessary, no distortion, notannoying appreciable effort required 3 Fair perceptible, slightlymoderate effort annoying required 2 Poor annoying but not considerableeffort objectionable required 1 Unsatisfactory very annoying, no meaningunder- objectionable stood with any reason- able effort

Local telephone service is generally considered to have a MOS score of4.5. A very good digital wireless call with a high-signal-to-noise ratio(CDMA or GSM) typically scores a MOS of 3.0-3.5. The two codecs used bymost often in VoIP networks, G.711 and G.729a, have MOS scores ofapproximately 4.4 and 3.9 respectively.

One limitation with MOS is that it can neither be applied on a widescale nor used on a daily basis to evaluate network call quality. Analternative, Perceptual Speech Quality Measures (PSQM) is an objectiveapproach to measure the quality of a telephone call and is based on ITUstandard P.861. PSQM defines an algorithm through which a computer canderive scores based on levels of distortions to a sound file between thesent and received audio tracks. PSQM, which uses an inverted scale fromMOS, provides a reasonably close correlation to MOS, with the limitationthat PSQM was not originally designed for packet telephony networks andtherefore only partially accounts for packet loss and jitter. DespitePSQM's limitations, it remains the method of choice because it isquantitative, repeatable and scalable. Thus, it is a preferred scale foruse in conjunction with certain embodiments. The PSQM scale issummarized in TABLE 2 below: TABLE 2 PSQM Type of Call 1.15 Local PSTNcall 1.20 G.711 codec 2.40 G.729a codec 3.20 Call with slightlyperceptible audio loss 3.50 Good digital mobile call

To further simplify understanding of these scores, in certainembodiments, the scores can be equated to display colors and alertlevels when the scores are displayed numerically or graphically. In oneembodiment, a “yellow alert” threshold can be set at a quality score ofapproximately 2.8 (e.g., in the range of approximately 2.5 to 3.1) toindicate that call quality may be beginning to deteriorate. A secondthreshold designate “red alert” can be set at a higher threshold ofdegradation, say approximately 3.5 (e.g., approximately 3.3 to 3.7) toindicate that call quality has deteriorated to a degree that will benoticed by a customer.

To determine the quality formula to be used by stream quality analyzer530, independent measurements of voice quality on the subject VoIPnetwork are made. These measurements may be subjective (e.g., ratings byhuman listeners) and/or objective (e.g., scores produced by computerizedvoice quality testers). Simultaneous samples of network metrics (jitter,packet loss, latency, etc.) are taken by the CQA for the same sessionsor links being independently observed and rated. This results in networkmetrics that are correlated to the voice quality scores resulting fromthe independent measurements.

With sets of correlated data in hand, relationships between theindependently measured quality scores and the network metrics arestudied in order to determine a function of the available networkmetrics that best matches the output of the independent measurements.Data fitting techniques such as linear regression and curve fitting orany other suitable data fitting technique may be utilized to determinethe quality equation used to fit the network metrics to perceived voicequality. Ideally, quality measurements may be taken in which all but oneof the correlated metrics are relatively constant, permitting study ofthe relationship between voice quality and the variable metric inisolation, but this is often difficult or impossible. A variety oftechniques may be used to produce a formula from the correlated datathat offers results of the desired accuracy and precision: linearregression analysis, curve fitting, graphing, etc. Since therelationships between quality scores and network metrics aremulti-variate and often non-linear, and since the data source used bythe CQA may not provide all the relevant data required to fullycharacterize voice quality, a series of judicious guesses may be triedto determine the formula with an optimal fit to empirical measurements.

Once the basic formula has been calibrated against independentmeasurements of voice quality, the range and distribution of qualityscores generated by the formula may be modified to correspond to anycommonly used voice scoring system (such as MOS, PSQM, PESQ, MNB, Rfactor or any other known or newly devised scoring system), so that themeaning of the numerical scores will be evident to those familiar withthe particular scoring system chosen. In order to keep the relationshipto empirical measurements intact, care should be taken during theconversion to maintain numerical correlations between checkpoints in thesource and destination scoring systems (that is, scores in the twosystems that are recognized as applying to similar conditions).

The overall process is described as discussed above is shown as process600 of FIG. 6, starting at 604. At 608, quality benchmarks areestablished using audio listening tests for calibration against aquality score Q. This testing may only need to be done once to establishthresholds and metrics for the relationship between the quality score Qand actual perceived voice quality. However, the testing may be anongoing process of refinement and may be needed when changes are made inthe network. The benchmarks are established against a quality score todevise a formula at 612 that correlates with the observed voice qualitydegradation in the face of various packet anomalies jitter, packet loss,latency, etc.), and thus, the process of 608 and the process of 612 maybe intimately related and may lend itself to an iterative process forestablishing benchmarks and refinement of the quality score.

Thus, the quality formula is developed by matching observed quality forcommunications over the IP network to a standard quality measurementscale, and equating the observed quality to the quality score using atleast several relevant ones of the available quality metrics (at leasttwo metrics). This is done by varying the metrics (preferablyindependently) and observing the effect on perceived quality.

Actual call data are sampled at 616 from the data stream representing aVoIP call by the stream collector 504 at 616. This data can be collectedat any number of points along the call path to help identify sources ofproblems, but is preferably collected at the CODEC output (i.e.,generally at the switch) to provide an end to end measurement. Thestream collector 504 then generates the metrics of packet loss, andtiming related metrics such as latency and jitter from the samples at620. If other metrics are available at 624, they are also collected at628. Whether or not other metrics are available, all available anduseful or significant metrics are collected and used in the qualityscore formula at 632 in order to output the quality score at 636.

Using this process, the actual quality score for a given sample of asession is computed and potentially available for use in any number ofmanners in near real time (i.e., within a second or two of the actualsamples) at this point. However, it has been found useful to actuallybuild in delays into the process to avoid overwhelming system resources.Also, it is useful to aggregate the results of several samples at 640prior to use for certain applications (e.g., simply displaying theresults) in order to improve the accuracy of the quality scores. Thescore or scores can then be sent to display, alarm, control and/orstorage functions at 644, so that the score and/or aggregations ofscores can be displayed, stored in a database and/or used to controlvarious network devices. In certain embodiments, scores can be updatedevery one to two minutes (still within the realm of near real time inthis application since actions can often be taken preemptively beforeany service quality degradation is perceived by the customer). Thisprovides, in certain embodiments, the ability to alert prior to theend-user perceiving a degradation in voice quality.

The same method may be used to generate quality scores for non-voiceaudio, video, fax, or other forms of telecommunication over apacket-switched network. Many other embodiments are possible within thescope of one of ordinary skill in the art, in view of the presentteaching.

Thus, a method for near real time quality analysis consistent withcertain embodiments passively samples packets from a stream of InternetProtocol (IP) packets that represent a communication session between apair of end points carrying analog signals being transmitted over atransmission path in an IP network, and determines in near real time atleast two metrics from the sampled packets for the communicationsession. The at least two metrics include at least one metric thatmeasures a quantity of lost packets, and at least one metric thatmeasures a characteristic of packet timing. The method further involvescalculating a quality score in near real time using a quality formulathat combines the at least two metrics.

Thus, the process takes real time samples of embedded packets andapplies an algorithm which emulates the analog qualities as experiencedby the human ear or eye (listener or viewer). This information can thenbe used to manage and troubleshoot data infrastructures being used forvoice and video applications.

If one believes that in order to realize the true value of IPT, it isbest to deliver a full IPT implementation as close to the caller anddesktop computer as possible, then one should also acknowledge that theservice provider network now includes network elements both within theircontrolled environment and outside of it effectively eliminating thedemarcation point. Customer local area network (LAN) components then,such as routers, switches, hubs and firewalls can have a deleteriouseffect on voice quality, so much so that measuring and testing calls toinclude these components should become a service component.

One of the real world difficult aspects of implementing a call qualityanalysis system and method consistent with certain embodiments herein,is development of the appropriate formulae for characterizing the callquality based upon the available metrics. While any available metricsmay be useful in arriving at a call quality score (Q), the metrics ofjitter, packet loss and latency are readily measured using RTP data asdescribed above. Several generalities can perhaps be made, but onlyafter observation of significant numbers of network configurations andgeneration of significant numbers of quality score equations.Additionally, the quality score can be made to match to any number ofstandard quality scores such as PSQM or PESQ, further rendering theactual process difficult to define.

Examples are illustrative, but the reader is cautioned that what appearto be generalities in the example below could prove erroneous whenlarger numbers of networks are analyzed.

In the present exemplary embodiment a real-time view of quality of datastreams is provided on a packet-switched network. In this exemplaryapplication, the data streams are Voice-over-IP (VoIP) calls on a TCP/IPnetwork, and that will be the primary context for exploring thisexample. In this example, calls are placed over a network having local,LAN, WAN and Internet transport media. However, the same techniques aremore broadly applicable, such as to streaming video, Fax-over-IP, orother forms of communication which are dependent upon the synchronizeddelivery of data packets in real time.

For this example, a “stream” is hereby defined as a sequence of datapackets sent over a network from a source computing device to one ormore destination computing devices, where the contents of the packetsare presented to the user of the destination device in a particularorder and in a manner which appears continuous to the user. Normally thepacket contains digitized representations of analog information: sound,pictures, video, etc. The user-perceived quality of such a stream (suchas vocal fidelity for a VoIP call, or image and sound quality for astreamed movie), as previously discussed, is potentially degraded whenpackets are lost, arrive out of order, are delayed, or experiencevariable transit times, thus affecting the smoothly continuouspresentation of data to the user, as described above.

In order to calculate a quality factor, somewhat more detaileddefinitions are useful. Following are more mathematical definitions ofthe metrics described above that are used to calculate stream quality inthe current example:

Packet Loss=P and is defined in terms of a ratio of number of packetsthat never arrive at the destination, or arrive later than somepredefined interval, to the number of packets produced at the source.

Latency=L and is the amount of time it takes a packet to travel fromsource to destination.

Jitter=J and is a measure of the variation in latency between packets.If every packet takes the same amount of time to transit the network(i.e. has the same latency), jitter is zero. The greater the variabilityin transit times, the higher the jitter.

Referring back to FIG. 5, the passive stream collector 504 isimplemented in this example as software residing on one or more computersystems on a network. The purpose of the passive stream collector 504 isto directly sample the packets traversing the network, identify datastreams, and determine raw metrics (such as packet loss, latency andjitter) which can be used to determine quality of the streams. In orderto perform this function, the system on which the passive streamcollector 504 software resides has access to the communications on thenetwork. However, passive stream collector 504 is a passive element onthe network, and inspects the network traffic without interfering withit.

Stream-based IP packet communications utilizes the Real Time Protocol(RTP) to carry the digitized content of the communication. Setup andteardown of the stream is handled by one of a number of other industrystandard protocols (H.323, SIP, MGCP, MEGACO, Skinny, etc.), known assignaling protocols. The passive stream collector 504 processessignaling traffic, which is used to identify which RTP packets belong towhich streams. Passive stream collector 504 then processes the RTPtraffic to determine quality metrics.

For each RTP stream (which comprises one leg of a VoIP call, streamedvideo, etc.), the passive stream collector 504 examines a number ofpackets in sequence. In order to calculate stream quality, it is notnecessary for the passive stream collector 504 to examine every packetin a stream (but this could also be done); it may take periodic samplesof sequential packets, with the period adjustable for how frequentlyresults are desired, the typical interval between packets, thecapability of the computer system on which the software runs to processthe packets, etc. In the current example, the inter-packet interval is20 ms, and 16 packets are sampled at a time about 3 times per minute.The sampling rate and size are easily adjustable.

As examples of how the metrics are calculated, packet loss is calculatedby looking at the RTP sequence number in each packet. Gaps in thesequence represent lost packets, and the packet loss is calculated asthe number of lost packets divided by the sum of received packets pluslost packets. Latency is calculated by looking at the RTP timestamp ineach packet, and comparing it to the time on the passive streamcollector 504 to determine transit time. This requires precise timesynchronization between the source and the passive stream collector 504.Latency for the sample is the average of the latencies of the packets.Jitter is calculated by computing latency as above for each packet, thenusing the differences between these latencies in a standard formulagiven by Internet Engineering Task Force (IETF) Request for Comments(RFC) 1889. Since the time calculated for each packet is relative tothose for the packets immediately preceding and following, precisesynchronization between source and destination is not necessary(although abrupt time changes, such as for Daylight Savings Time, mustbe taken into account). Once metrics such as these are calculated for asample, they can be used in a quality formula to determine a qualityscore for the sample.

The Stream Quality Analyzer (SQA) 530 is also implemented in thisexample as software residing on one or more computer systems on anetwork (perhaps the same computer as the passive stream collector 504,or perhaps a central computer system receiving metrics from multiplestream quality analyzers 530). The purpose of the SQA is to calculateindividual quality scores for communications streams on the network.Data used in the calculation may come from a number of differentsources, including (but not limited to): samples of networkcommunications, in which digitized voice media is typically carried inRTP packets, as discussed above; metrics contained within Real TimeControl Protocol (RTCP) packets associated with a stream; NPTP resultsrecorded on network devices such as VoIP gateways; call metrics storedon a softswitch or other VoIP network element; call metrics stored on ahandset or other end-device; and/or other metrics available from networkdevices.

One or more sources of data may be used, depending on what is availableon the particular network. The mathematical quality formula used tocalculate the quality score is derived by comparing the values of theraw data for a stream to independent measures of the stream's quality,and constructing a formula that reproduces the independently determinedquality scores as closely as possible given the correlated raw data.Independent quality measurements may be subjective (such as a MeanOpinion. Score (MOS) given to a VoIP call by a panel of human listeners)and/or objective (such as a Perceptual. Speech Quality Measure (PSQM)score generated by a computerized VoIP tester).

With sets of correlated data in hand, relationships between theindependently measured quality scores and the network metrics arestudied in order to determine a function of the available networkmetrics that best matches the output of the independent measurements.Ideally, quality measurements may be taken in which all but one of thecorrelated metrics are relatively constant, permitting study of therelationship between stream quality and the variable metric inisolation, but often this is not the case—rendering the refinement ofthe equation tricky. A variety of techniques may be used to produce aformula from the correlated data that offers results of the desiredaccuracy and precision: linear regression analysis, graphing, etc. Sincethe relationships between quality scores and network metrics aretypically multi-variate and non-linear, and since the data source usedby the SQA may not provide all the relevant data required to fullycharacterize stream quality, it is often necessary to try a series ofjudicious guesses to determine the formula with the optimal fit toempirical measurements. For reference, in the current exemplaryembodiment, the SQA formula makes use of both logarithmic andexponential functions (both of which are transcendental, non-linearfunctions), along with empirically determined coefficients, addends, andfunction arguments.

It is possible and in fact quite likely that a quality formula that iswell suited to one network environment may not be applicable to another,so this general procedure may need to be repeated in whole or in partfor each new environment (possibly including changes to the existingenvironment). Factors that distinguish network environments includelogical and physical placement of network elements, signaling protocolsused, intervals between RTP packets, voice or video codecs, etc.

Once the formula has been determined, the SQA 530 may continuouslymonitor network traffic, automatically generating quality scores forVoIP calls or other multimedia data such as streamed movies, faxes, etc.

The quality scores generated by the SQA may be used for a variety ofpurposes. For instance, they may be formatted for display in a dedicatedweb interface (known as a Stream Quality Display, or SQD), monitored bynetwork support personnel to be made aware of situations adverselyaffecting stream quality. The quality data may be formatted in a numberof different ways: as numerical scores, possibly matching the range ofan industry-standard measure of quality (MOS, PSQM, PESQ, MNB, R factor,etc.), colors (green, yellow, red, etc.), letter grades (A, B, C, etc.),etc.

The quality scores may be fed into an automated system that generatesalarms or alerts to network managers when quality degrades beyond aspecified threshold. Quality data may be integrated into an existingnetwork management system (NMS) for display and alerting. Given aproperly constructed network architecture, quality data could be used toautomatically modify the routing of data packets in near-real time towork around network links experiencing transient quality degradations,thus improving the end-user experience. Quality data could also be usedas metrics to judge compliance with a service level agreement (SLA)guaranteeing minimum levels of quality for network services.

The scores themselves are essentially arbitrary, so once in service it'sdifficult to distinguish between the need to tweak the function andsimply adjusting perceptions of what the numbers mean. The best waycurrently known to determine the function is to correlate the metricswith other quality measures. The function can be fine-tuned bycontinuing the process with more data.

As a guide to developing the stream quality formula, following is adiscussion of the general effects on quality of each of the metricspreviously mentioned. They cannot be generalized in a precise fashion,since their influences can be modified by many factors, including:

Different codecs may be more or less susceptible to quality degradationby jitter, latency, or packet loss.

Devices participating in data transfer may compensate for packet loss byfilling in gaps with the use of interpolation algorithms (whichthemselves may do better or worse in different situations).

Devices participating in data transfer may compensate for jitter withthe use of a “jitter buffer”, holding packets in memory for a short timeand playing them back out at regular intervals in order to reduce jitter(and packet loss, by allowing extra time for arrival) at the cost ofincreased latency.

Depending on where the metrics used by the SQA are obtained in thenetwork, it may or may not be able to directly observe the effects ofthe compensatory mechanisms used by certain devices. For example, if theSQA is utilizing data obtained by the quality stream collector 504 on aVoIP network, the packets analyzed by the quality stream collector 504may not be affected by gap-filling or jitter buffering taking place on aPSTN gateway or an IP phone. The particular application may dictatewhether these expected effects should attempt to be taken into accountwhen constructing the formula, or whether they should be ignored inorder to gauge the performance of the network in isolation.

The formula may be constructed to output quality scores suitable forwhichever scoring system is desired. In the VoIP industry, there areseveral systems in use: MOS, PSQM, PESQ, MNB, EMBSD, R Factor, etc. Someof these systems (MOS, PESQ, MNB, R Factor) are “positively aligned”,meaning that higher scores indicate better quality; others (PSQM, EMBSD)are “negatively aligned”, meaning that higher scores indicate worsequality.

Contrary to conventional wisdom that says packet loss or latency is theprimary contributor to degraded packet voice quality, jitter appears tobe the most significant variable in real life networks examined thusfar. In experimental voice quality models, over 90% of the change invoice quality is generally a function of the voice packet arrivalvariability (jitter). Fixed latency does not cause significantlydegraded voice quality unless the latency during a conversation exceedsthe International Telecommunication Union's (ITU) recommendation of 150ms one-way latency. Further, packet loss, particularly with the G.729acodec which was specifically designed to preserve clarity in high packetloss environments, will have unnoticeable changes in voice up to 5%packet loss, after which is quality degrades. Other codecs, such asG.711, break down rapidly in networks exceeding 1% packet loss. For thisreason, the quality score is often dominated in many regions by jitter.

Packet loss, when it occurs, can have a dramatic effect on quality.However, device characteristics such as those mentioned earlier oftencompensate for small amounts of loss.

For this reason, subtracting (or adding, in the case of negativelyaligned scoring systems) a term that varies exponentially with packetloss may work well, as may piecewise linear step and/or rampapproximations with coefficients chosen to scale to the appropriaterange of scores. If the effects of device characteristics are to beignored, then it may be more appropriate to use packet loss as amultiplicative factor for the formula as a whole.

Latency generally has a very small effect on quality, until it exceeds acertain threshold, or “latency budget” (generally considered to beapproximately 150 ms for VoIP networks). Therefore, an exponential termor piecewise linear step and/or ramp approximations may also beappropriate for latency, with very little contribution under 150 ms, andincreasingly greater contribution over that. Alternatively, another termthat only starts meaningfully contributing after a threshold of latencycan be used.

Experience has shown that under normal circumstances (little or nopacket loss, latency within budget), jitter is the primary contributorto quality degradation. Even in situations of high latency, jitter tendsto dominate. Jitter is usually strongly correlated with latency, makinglatency more an easily measurable design constraint than a primarymetric for use in quality determination. As with packet loss, smallamounts of jitter are often compensated for by devices on the network.

Practice has so far suggested that subtracting (or adding, in the caseof negatively aligned scoring systems) a component that starts small,rises fairly quickly with increasing jitter, but with a decreasing rateof increase, seems to work well in modeling the effect of jitter. Thus,using jitter in a logarithmic, root (such as a cube root), piecewiselinear approximations or even hyperbolic tangent term may beappropriate.

As one example, a quality formula taking the following form was used:Q=K ₁+ln(K ₂ +K ₃ J)+exp(K ₄ P),  (1)where the K values are empirically determined constants, where Q is thequality score, J is jitter (in milliseconds), and P is packet loss(number ranging from 0 to 1).

A quality formula following this general form was used in a VoIP networkenvironment where jitter and packet loss were the available metrics, andthe output was scaled as PSQM scores. The network topology includedCisco 7960 and 7910 phones connected via 10/100 Ethernet ports to aCisco 3524 switch with line power cards. All voice traffic is sent outof a gateway at the customer site via a T1 line to Voice Firewallslocated at a specified POP. The VoIP signaling is relayed through theVoice Firewalls to the Call Agent Server at a data center. Calls arethen completed via the appropriate POP's Cisco 5300, 5400 or Vocal Datagateways or a destination IP phone The formula was constructed byempirically adjusting the equation for a close match to actual data aswill be explained, resulting in:Q=1.4+ln(1+0.17J)+exp(1.1P).  (2)

The initial term, when combined with the value of the exponential termunder the usual condition of P=0, is used to produce a minimum value of2.4, the best possible PSQM score in a VoIP network environmentutilizing the G.729 codec. The middle term is zero when J=0, but resultsin a score of about 3.4 (perceptible distortion) when J=10, and a scoreof about 4.6 (nearly unintelligible) when J=50, which is equal to thenormal inter-packet spacing of 20 ms on this network. The final term hasa very small contribution up to about P=0.2 (20% packet loss), butincreases quickly.

As a second example, an equation having the general form that followswas used:Q=K ₅ −K ₆ L+K ₇ R+K ₈ J,  (3)where Q is the quality score, L is latency (in milliseconds), R is thesum of the squares of round-trip times (where round-trip time, or RTT,is the combined latency for transit from source to destination andback), and J is the minimum positive jitter (in milliseconds).

Again, this formula was matched to observed data and the followingquality formula was used in a VoIP network environment where a varietyof calculated metrics related to latency and jitter were available, andthe output was scaled as PSQM scores:Q=2.411−0.0105L+0.000006382R+0.0237J.  (4)

This formula was derived from a linear regression analysis of theavailable data. The negative sign before the latency term may seemsurprising, but it should be recalled that the variables used in thisformula are not truly independent, and the latency term may serve tocorrect for the effects of the RTT or jitter terms.

From the above examples, it can be seen that it is difficult, if notimpossible, to generalize the formula used by the stream qualityanalyzer 530 to generate an appropriate quality score for any givennetwork with any specifics. These examples only use packet loss metricsand packet timing metrics, but in general, other quality metricsavailable from network devices should be useable to further refine anestimate of quality. It can be generally stated that the quality scorecan be defined as:Q=f(packet timing, packet loss, other quality metrics).  (5)

Thus, it should be noted that the above examples and guidelines arebased upon a very small sample of networks, and therefore some of thesuggestions given above for constructing the formula could be completelywrong for different environments. While the above guidelines may bevalid, the possibility that they are in error for certain networksshould be born in mind when developing a quality score formula. Once anappropriate formula is devised, basic metrics can be measured inreal-time, with the formula used to generate quality scores based onthose metrics in real-time, for use in a variety of contexts.

In order to track each end-point and customer for the presentation ofperformance data, a database can be used that links MAC addresses tosome meaningful information such as:

CustomerCompany:CustomerSite:User:MAC.

or

TelephoneNumber:User:MAC

or

End User Name: User: MAC.

In certain embodiments a web interface is used for viewing the qualityscore output which can be viewed using a suitable browser applicationsuch as Microsoft Internet Explorer™, and quality scores are updated ata reasonable periodic rate such as one minute intervals. A referencetable that maps device MAC address to username and telephone number (orsome other unique identifying information other than MAC address) can beused to allow a clean, user friendly interface for quality scoreresults.

The PSQM scores, according to the present embodiment are color coded(for easy monitoring) as follows: TABLE 3 Minimum Score Maximum ScoreColor (Less than or equal to) (Greater than) Green 0 2.8 Yellow 2.8 3.5Red 3.5 3.5<

A Quality Score is devised to track the numerical PSQM with a scoringmethod that is easily understood by the general customer for reportsprimarily reviewed by end user customers. Several options are possible,such as a traditional grading score (A, B, C, D or F) or simple colorcoding (green, yellow or red). In other embodiments, the real PSQMscores may be hidden based on a browser cookie setting so that certainusers such as engineers can see the numeric values and other users suchas customers will only see colors.

Referring now to FIG. 7, a user interface screen 700 is depicted for aconfiguration manager screen. On this screen, basic configurationinformation can be set globally and based on customer. When the softwareis loaded by linking to an IP address or web address in a conventionalmanner, the configuration manager screen can be called by selecting theconfiguration option 702. The display then shows the configuration for adesignated customer at 704. Various configuration parameters can then beobserved by name in column 706 with the respective value for anassociated parameter in column 708. Actions of either obtaining help asillustrated or editing a value can be carried out by selection of theappropriate icons in column 710. Of significance, in this screen, thequality scores are refreshed every 60 seconds (refreshinterval) and thered alert and yellow alert thresholds are set as per TABLE 3. Thesethresholds are used globally throughout the system and can be setmanually or dynamically with the software responding appropriately toany changes.

With reference to FIG. 8, the screen shot 716 depicts the qualityscoring results starting at a high level overview called CustomerSummary 718. This web page presents an overall score for each customerin column 720 presented in different hourly increments in column 722.The most recent score is in column 724. This view is intended to providewith a high level overview to facilitate quickly identifying networkproblems for various customers. Active and inactive elements areidentified in columns 726 and 728. Column 730 can be used to edit thethreshold values for a given customer. The background (or foreground)color used in the cells of the table on this screen shot can be colorcoded to the alert level corresponding to the quality score depicted.

Clicking on the company's name in the Customer Summary screen or on themenu bar calls up the Customer Detail page as depicted in FIG. 9. Onthis page, the customer is identified at 738 and each customer device asidentified in column 740. Each end-point in the Customer Detail matrixis a link to a page devoted to historical data relating to that device.Scores for connections to each device identified by IP address are shownin columns 742 and 744. Column 746 is again used to edit thresholds. Thebackground (or foreground) color used in the cells of the table on thisscreen shot can be color coded to the alert level corresponding to thequality score depicted.

Columns 742 and 744 are Voice Firewalls and are used to show the matrixof scores between the various end devices (rows illustrated by 740) andthe voice firewalls 742 and 744. Although not shown explicitly in thisillustration, the column headings for 742 and 744 can match the columnheadings (or a sub-set thereof) of those illustrated in 792 of screenshot 786 shown later. The present screen shot is a more granular look athow each end-device is routing (and the quality of that routing) to thevarious voice firewalls, or what is referred to as the VoIP Proxy in thesoftware.

Each cell in columns 742 and 744 shows a quality score associated withrecent sessions between the two end points that define the intersectionof the cell along with the color grade associated with the qualityscore. In certain embodiments, the score displayed can be an average ofall scores calculated for the pair of endpoints over a designated timeperiod (e.g., the past five minutes, with the average recalculated everyone minute). This time period can be made adjustable or fixed so that alonger period of average scores is more indicative of overall quality ofconnection between the end points, while a shorter time period is moreindicative of short term or instantaneous quality for the connectionbetween the end points. Of course, short term scores are more erraticthan longer term scores due to variations resulting from networkvariations. If no score has been calculated for a designated pair of endpoints, the cell can be left blank or a historical value can bedisplayed with a designation as such.

By selecting a particular device, the user can bring up screen 750 asdepicted in FIG. 10. In this screen, the device history is graphicallydepicted in chart 752 with the device identified by coded device name754 as well as a more user friendly identifier at 756. Preferably, thischart is color coded so that the bars of the graph are shown in greenwhen below the yellow alert threshold 758 (e.g., at 760), yellow betweenthe yellow alert threshold 758 and the red alert threshold 762 (e.g., at764), and red above the red alert threshold 762 (e.g., at 766). Thesecolor codes are given meaning in the key at 768. The color thresholdsestablished and configured in the configuration manager are the samethresholds used here and are ultimately tied together.

Additional historical detail can also be obtained as depicted in thescreen shot 772 of FIG. 11. In this screen, the device is identified byIP address at 774 and by a more easily recognizable name at 776. Thehistorical data are represented again using colors in graph 778. Again,data points are color coded as in the prior figure according to theirrelationship to the threshold levels 758 and 762, and the colors arekeyed at 780. In this graph, a trailing high score is also shown in thesolid curve 782

With reference to FIG. 12, by selecting the POP summary from the menubar at 788, a summary of the current Point Of Presence can be viewedwith identifying information for the POP in column 790 with scoresbetween each voice firewall and an associated gateway appearing incolumns such as 792, preferably color coded according to score.

With reference to FIG. 13, upon initial loading of the software, ornegotiation to a home page 804 using the menu bar at 794 or help page,the user may be greeted with an explanatory brief on the content of eachtype of screen. Row 796 explains the POP summary page 786; row 798explains customer summary page 716; row 800 explains customer detailpage 736; and row 802 explains where other application documentation canbe found.

Those skilled in the art will appreciate upon consideration of thisdiscussion of the user interface, that many other user interfaces andvariations on the present interface are possible without departing fromembodiments consistent with the present invention. Accordingly, theseillustrative examples are intended to illustrate one technique forpresenting a user interface to the quality analysis system disclosedherein without limitation.

While the present discussion has centered around use of the presenttechnology for VoIP applications, the present techniques and systems areequally applicable to other multimedia applications without limitation.It is further noted that while the present techniques have beendeveloped for analysis of information that originates and terminates asanalog information, these techniques are not limited to such signals.For example, digital multimedia files transmitted over similar networkscan be analyzed in similar ways without departing from embodimentsconsistent with the present invention.

Those skilled in the art will recognize, upon consideration of the aboveteachings, that certain of the above exemplary embodiments are basedupon use of a programmed processor, e.g., to implement the CQA. However,the invention is not limited to such exemplary embodiments, since otherembodiments could be implemented using hardware component equivalentssuch as special purpose hardware and/or dedicated processors. Similarly,general purpose computers, microprocessor based computers,micro-controllers, optical computers, analog computers, dedicatedprocessors, application specific circuits and/or dedicated hard wiredlogic may be used to construct alternative equivalent embodiments.

Certain embodiments described herein, are or may be implemented using aprogrammed processor executing programming instructions that are broadlydescribed above that can be stored on any suitable electronic orcomputer readable storage medium and/or can be transmitted over anysuitable electronic communication medium. However, those skilled in theart will appreciate, upon consideration of the present teaching, thatthe processes described above can be implemented in any number ofvariations and in many suitable programming languages without departingfrom embodiments of the present invention. For example, the order ofcertain operations carried out can often be varied, additionaloperations can be added or operations can be deleted without departingfrom certain embodiments of the invention. Error trapping can be addedand/or enhanced and variations can be made in user interface andinformation presentation without departing from certain embodiments ofthe present invention. Such variations are contemplated and consideredequivalent.

Software and/or firmware embodiments may be implemented using aprogrammed processor executing programming instructions that in certaininstances are broadly described above in flow chart form that can bestored on any suitable electronic or computer readable storage medium(such as, for example, disc storage, Read Only Memory (ROM) devices,Random Access Memory (RAM) devices, network memory devices, opticalstorage elements, magnetic storage elements, magneto-optical storageelements, flash memory, core memory and/or other equivalent volatile andnon-volatile storage technologies) and/or can be transmitted over anysuitable electronic communication medium. However, those skilled in theart will appreciate, upon consideration of the present teaching, thatthe processes described above can be implemented in any number ofvariations and in many suitable programming languages without departingfrom embodiments of the present invention. For example, the order ofcertain operations carried out can often be varied, additionaloperations can be added or operations can be deleted without departingfrom certain embodiments of the invention. Error trapping can be addedand/or enhanced and variations can be made in user interface andinformation presentation without departing from certain embodiments ofthe present invention. Such variations are contemplated and consideredequivalent.

While certain illustrative embodiments have been described, it isevident that many alternatives, modifications, permutations andvariations will become apparent to those skilled in the art in light ofthe foregoing description.

1. A near real time quality analyzer, comprising: a passive streamcollector that passively samples packets from a stream of InternetProtocol (IP) packets that represent a communication session between apair of end points carrying analog signals being transmitted over atransmission path in an IP network, and determines in near real time atleast two metrics from the sampled packets for the communicationsession; wherein the at least two metrics comprise: at least one metricthat measures a quantity of lost packets; and at least one metric thatmeasures a characteristic of packet timing; and a stream qualityanalyzer that receives the at least two metrics and calculates a qualityscore in near real time using a quality formula that combines the atleast two metrics.
 2. The near real time quality analyzer according toclaim 1, wherein the at least one metric that measures a characteristicof packet timing measures at least one of packet jitter, packet latency,and round trip time.
 3. The near real time quality analyzer according toclaim 1, wherein the passive stream collector samples packets at aswitch.
 4. The near real time quality analyzer according to claim 1,wherein the passive stream collector samples all packets entering andleaving a switch.
 5. The near real time quality analyzer according toclaim 1, wherein the stream quality analyzer receives additional metricsfrom network devices residing in the IP network along the transmissionpath, and wherein the quality formula incorporates functions of theadditional metrics.
 6. The near real time quality analyzer according toclaim 5, wherein the additional metrics comprise at least one of a softswitch call metric, call metrics stored on an end-device, a VoIP (Voiceover IP) network component, and a Network Performance Test Probe (NPTP)result.
 7. The near real time quality analyzer according to claim 1,wherein the Internet Protocol (IP) packets that represent analog voicesignals in which digitized voice is carried in Real Time Protocol (RTP)packets.
 8. The near real time quality analyzer according to claim 1,wherein the at least two metrics are derived from data within Real TimeControl Protocol (RTCP) packets.
 9. The near real time quality analyzeraccording to claim 1, wherein the quality score is stored in a databaseindexed to the pair of end points.
 10. The near real time qualityanalyzer according to claim 1, further comprising means for generatingan alarm whenever the quality score falls below a quality threshold. 11.The near real time quality analyzer according to claim 1, furthercomprising a display wherein the quality score is displayed on thedisplay.
 12. The near real time quality analyzer according to claim 11,wherein the display further displays historical quality scoresassociated with the end points.
 13. The near real time quality analyzeraccording to claim 1, wherein the stream quality analyzer aggregates aplurality of quality scores.
 14. The near real time quality analyzeraccording to claim 1, wherein the passive stream collector and thestream quality analyzer are implemented as programmed processes on acomputer.
 15. The near real time quality analyzer according to claim 1,wherein the quality formula takes the general form of:Q=K ₁+ln(K ₂ +K ₃ J)+exp(K ₄ P) where where Q is the quality score, theK values are constants, J is jitter, and P is packet loss.
 16. The nearreal time quality analyzer according to claim 1, wherein the qualityformula takes the general form of:Q=K ₅ −K ₆ L+K ₇ R+K ₈ J where Q is the quality score, the K values areconstants, L is latency, R is the sum of the squares of round-triptimes, where round-trip time is the combined latency for transit betweenthe pair of end points, and J is a minimum positive jitter.
 17. The nearreal time quality analyzer according to claim 1, wherein the qualityformula is developed by matching observed quality for communicationsover the IP network to a standard quality measurement scale, andequating the observed quality to the quality score as the at least twometrics are varied.
 18. The near real time quality analyzer according toclaim 1, wherein the quality formula is designed to approximate aPerceptual Speech Quality Measurement (PSQM) score.
 19. The near realtime quality analyzer according to claim 18, wherein an alert thresholdlevel is defined for quality scores of approximately 3.5.
 20. The nearreal time quality analyzer according to claim 18, wherein an alertthreshold level is defined for quality scores of approximately 3.3 to3.7
 21. The near real time quality analyzer according to claim 18,wherein an alert threshold level is defined for quality scores ofapproximately 2.8.
 22. The near real time quality analyzer according toclaim 18, wherein an alert threshold level is defined for quality scoresof approximately 2.5 to 3.1.
 23. The near real time quality analyzeraccording to claim 18, wherein a low level alert threshold level isdefined for quality scores of approximately 2.8, and wherein a higherlevel alert threshold level is defined for quality scores ofapproximately 3.5.
 24. The near real time quality analyzer according toclaim 18, wherein quality scores exceeding the low level alert thresholdlevel but not the higher level alert threshold are color coded as yellowquality level, and wherein quality scores exceeding the higher levelalert threshold level are color coded as red quality level, and whereinquality scores lower than the low level alert threshold is color codedas a green quality level.
 25. The near real time quality analyzeraccording to claim 1, wherein: a low level alert threshold and a highlevel alert threshold are established, and wherein, quality scoresexceeding the low level alert threshold level but not the higher levelalert threshold are color coded as yellow quality level, and whereinquality scores exceeding the higher level alert threshold level arecolor coded as red quality level, and wherein quality scores lower thanthe low level alert threshold is color coded as a green quality level.26. The near real time quality analyzer according to claim 25, furthercomprising means for displaying the quality score on a display usingcolor codes for to indicate the quality score's relationship to thealert levels.
 27. The near real time quality analyzer according to claim25, wherein the quality score is compared to at least one threshold thatis established either manually or dynamically.
 28. A near real timequality analyzer, comprising: a passive stream collector that passivelysamples packets from a stream of Real Time Protocol (RTP) InternetProtocol (IP) packets entering and leaving a switch, wherein the streamof packets represents a communication session between a pair of endpoints carrying analog signals being transmitted over a transmissionpath in an IP network, and determines in near real time at least twometrics from the sampled packets for the communication session; whereinthe at least two metrics are derived from data contained within RealTime Control Protocol packets comprise: at least one metric thatmeasures a quantity of lost packets; and at least one metric thatmeasures a characteristic of packet timing, selected from the groupconsisting of packet jitter, packet latency, and round trip time; astream quality analyzer that receives the at least two metrics andcalculates a quality score in near real time using a quality formulathat combines the at least two metrics, wherein the stream qualityanalyzer aggregates a plurality of quality scores; a database, receivingthe quality score from the stream quality analyzer and storing thequality score indexed to the pair of end points; means for comparing thequality score with a quality threshold and generating an alarm wheneverthe quality score falls below a quality threshold; and a display thatdisplays the quality score along with historical quality scoresassociated with the end points.
 29. The near real time quality analyzeraccording to claim 28, wherein the stream quality analyzer receivesadditional metrics from network devices residing in the IP network alongthe transmission path, and wherein the quality formula incorporatesfunctions of the additional metrics.
 30. The near real time qualityanalyzer according to claim 28, wherein: a low level alert threshold anda high level alert threshold are established, and wherein, qualityscores exceeding the low level alert threshold level but not the higherlevel alert threshold are color coded as yellow quality level, andwherein quality scores exceeding the higher level alert threshold levelare color coded as red quality level, and wherein quality scores lowerthan the low level alert threshold is color coded as a green qualitylevel.
 31. A method for near real time quality analysis, comprising:passively sampling packets from a stream of Internet Protocol (IP)packets that represent a communication session between a pair of endpoints carrying analog signals being transmitted over a transmissionpath in an IP network, and determining in near real time at least twometrics from the sampled packets for the communication session; whereinthe at least two metrics comprise: at least one metric that measures aquantity of lost packets; and at least one metric that measures acharacteristic of packet timing; and calculating a quality score in nearreal time using a quality formula that combines the at least twometrics.
 32. The method for near real time quality analysis according toclaim 31, wherein the at least one metric that measures a characteristicof packet timing measures at least one of packet jitter, packet latency,and round trip time.
 33. The method for near real time quality analysisaccording to claim 31, wherein the packets are sampled at a switch. 34.The method for near real time quality analysis according to claim 31,wherein the samples are taken at a switch, and wherein the samples aretaken of all packets entering and leaving the switch.
 35. The method fornear real time quality analysis according to claim 31, furthercomprising receiving additional metrics from network devices residing inthe IP network along the transmission path, and wherein the qualityformula incorporates functions of the additional metrics.
 36. The methodfor near real time quality analysis according to claim 35, wherein theadditional metrics comprise at least one of a soft switch call metric,call metrics stored on an end-device a VoIP (Voice over IP) networkcomponent, and a Network Performance Test Probe NPTP result.
 37. Themethod for near real time quality analysis according to claim 31,wherein the Internet Protocol (IP) packets that represent analog voicesignals in which digitized voice is carried in Real Time Protocol (RTP)packets.
 38. The method for near real time quality analysis according toclaim 31, wherein the at least two metrics are derived from data in RealTime Control Protocol (RTCP) packets.
 39. The method for near real timequality analysis according to claim 31, further comprising storing thequality score in a database indexed to the pair of end points.
 40. Themethod for near real time quality analysis according to claim 31,further comprising generating an alarm whenever the quality score fallsbelow a quality threshold.
 41. The method for near real time qualityanalysis according to claim 31, further comprising displaying thequality score on a display.
 42. The method for near real time qualityanalysis according to claim 41, wherein the display further displayshistorical quality scores associated with the end points.
 43. The methodfor near real time quality analysis according to claim 31, furthercomprising aggregating a plurality of quality scores.
 44. The method fornear real time quality analysis according to claim 31, wherein theprocess is carried out on a programmed processor.
 45. The method fornear real time quality analysis according to claim 31, wherein thequality formula is developed by matching observed quality forcommunications over the IP network to a standard quality measurementscale, and equating the observed quality to the quality score as the atleast two metrics are varied.
 46. The method for near real time qualityanalysis according to claim 31, wherein: the quality formula is designedto approximate a Perceptual Speech Quality Measurement (PSQM) score, andwherein a low level alert threshold level is defined for quality scoresof approximately 2.8, and wherein a higher level alert threshold levelis defined for quality scores of approximately 3.5.
 47. The method fornear real time quality analysis according to claim 46, wherein qualityscores exceeding the low level alert threshold level but not the higherlevel alert threshold are color coded as yellow quality level, andwherein quality scores exceeding the higher level alert threshold levelare color coded as red quality level, and wherein quality scores lowerthan the low level alert threshold is color coded as a green qualitylevel.
 48. The method for near real time quality analysis according toclaim 31, wherein: a low level alert threshold and a high level alertthreshold are established, and wherein, quality scores exceeding the lowlevel alert threshold level but not the higher level alert threshold arecolor coded as yellow quality level, and wherein quality scoresexceeding the higher level alert threshold level are color coded as redquality level, and wherein quality scores lower than the low level alertthreshold is color coded as a green quality level.
 49. The method fornear real time quality analysis according to claim 48, furthercomprising displaying the quality score on a display using color codesfor to indicate the quality score's relationship to the alert levels.50. The method for near real time quality analysis according to claim31, wherein a packet latency metric is modeled in the quality formula aseither an exponential term or a piecewise linear function in which theoverall quality score shows a sharp decline in quality when packetlatency exceeds approximately 150 ms and a low effect on quality scorewhen packet latency is below approximately 150 ms.
 51. The method fornear real time quality analysis according to claim 31, wherein a packetloss metric is modeled in the quality formula as either an exponentialterm or a piecewise linear function in which the overall quality scoreshows a sharp decline in quality when packet loss exceeds a threshold.52. The method for near real time quality analysis according to claim31, wherein a packet latency metric is modeled in the quality formula asan overall formula multiplier.
 53. The method for near real time qualityanalysis according to claim 31, wherein a packet jitter metric ismodeled in the quality formula as either a logarithmic term, or ahyperbolic tangent function or as a piecewise linear function.
 54. Acomputer readable storage medium storing instructions which, whenexecuted on a programmed processor, carry out a process of near realtime quality analysis, comprising: passively sampling packets from astream of Internet Protocol (IP) packets that represent a communicationsession between a pair of end points carrying analog signals beingtransmitted over a transmission path in an IP network, and determiningin near real time at least two metrics from the sampled packets for thecommunication session; wherein the at least two metrics comprise: atleast one metric that measures a quantity of lost packets; and at leastone metric that measures a characteristic of packet timing; andcalculating a quality score in near real time using a quality formulathat combines the at least two metrics.
 55. The storage medium accordingto claim 54, wherein the at least one metric that measures acharacteristic of packet timing measures at least one of packet jitter,packet latency, and round trip time.
 56. The storage medium according toclaim 54, wherein the process further comprises storing the qualityscore in a database indexed to the pair of end points.
 57. The storagemedium according to claim 54, wherein the process further comprisesgenerating an alarm whenever the quality score falls below a qualitythreshold.
 58. The storage medium according to claim 54, wherein theprocess further comprises displaying the quality score on a display. 59.The storage medium according to claim 54, wherein the process furthercomprises aggregating a plurality of quality scores.
 60. A computerreadable storage medium storing instructions which, when executed on aprogrammed processor, carry out a process of near real time qualityanalysis, the instructions comprising: a first code segment thatpassively samples packets from a stream of Internet Protocol (IP)packets that represent a communication session between a pair of endpoints carrying analog signals being transmitted over a transmissionpath in an IP network, and determining in near real time at least twometrics from the sampled packets for the communication session; whereinthe at least two metrics comprise: at least one metric that measures aquantity of lost packets; and at least one metric that measures acharacteristic of packet timing; and a second code segment thatcalculates a quality score in near real time using a quality formulathat combines the at least two metrics.
 61. The storage medium accordingto claim 60, wherein the at least one metric that measures acharacteristic of packet timing measures at least one of packet jitter,packet latency, and round trip time.
 62. The storage medium according toclaim 60, further comprising a code segment that stores the qualityscore in a database indexed to the pair of end points.
 63. The storagemedium according to claim 60, further comprising a code segment thatgenerates an alarm whenever the quality score falls below a qualitythreshold.
 64. The storage medium according to claim 60, furthercomprising a code segment that aggregates a plurality of quality scores.